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Abstract 


Pre-college advising programs exist in most disadvantaged high schools throughout the 
United States. These programs supplement traditional advising by high school guidance 
counselors and attempt to help underrepresented and disadvantaged students overcome 
the complexities of the postsecondary admission and financial aid processes. Existing 
evidence on these programs often uses within-school randomization where spillovers 
and alternative supports may confound estimates. We provide the first evidence on a 
whole school intervention resulting from a school-level randomized controlled trial in 
the United States. The college access program we study uses a near-peer model where 
a recent college graduate works at the school assisting students in the application and 
enrollment process. Pooled results across the first three years of program implemen- 
tation find no significant impacts on overall college enrollment. However, subgroup 
analyses reveal positive, significant effects among the groups most targeted by the in- 
tervention: Hispanic and low-income students. Most of the impact comes through 
increasing two-year college enrollment, but this appears to be new entrants rather than 
inducing students to move from four-year to two-year colleges. The observed positive 
effects for these subgroups attenuate over time. We attribute this drop in the estimated 
impact to departures in fidelity of the experiment. Even among the cohorts for which we 
find positive enrollment impacts, we find no significant impacts on college persistence. 
© 2019 by the Association for Public Policy Analysis and Management. 


INTRODUCTION 


Improving college access and completion is critical to reducing economic inequality 
within the United States and to increasing the United States’ international compet- 
itiveness; yet planning for, applying to, attending, and succeeding in college are not 
easy for many families. Many well-qualified students are discouraged from pursuing 
higher education by avoidable barriers such as a lack of information about college 
admissions and financial aid (Avery & Kane, 2004). College advising is one of the 
key mechanisms by which policymakers, foundations, and high schools attempt to 
aid students as they navigate the college access “gauntlet” (Advisory Committee on 
Student Financial Assistance, 2005; Klasik, 2012), and across the country, there are 
thousands of college access programs that provide assistance to underserved stu- 
dents. These programs are so widespread that the umbrella National College Access 
Network claims the college access programs it represents provide supports to over 
two million students annually (National College Access Network, 2017). 
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The diversity of college access programs is staggering, even within the same school 
or community. These programs vary dramatically by their sponsoring organizations, 
funding sources, organizational structure, target populations, and interventions 
employed to improve college preparation and increase postsecondary enrollment 
(Gandara, 2001). A few of these programs operate nationally (e.g., Upward Bound, 
TRIO, and GEAR UP), but many are small and local, and therefore do not lend 
themselves well to rigorous evaluation and have limited external validity. 

Better understanding the magnitude of the effects of college access programs 
is essential as current federal budget proposals suggest slashing the federal in- 
vestment in TRIO and GEAR UP programs (a set of the most wide-scale access 
programs), which currently receive over a billion dollars annually from the federal 
government. Additionally, investments by school districts, states, and non-profit 
organizations in college access programs are substantial. The Gates, Kresge, and 
Lumina Foundations as well as many other national and local foundations have 
each devoted millions of private dollars to expand and improve these programs. 
Identifying causal effects of these programs at scale is essential for making in- 
formed policy decisions as policymakers seek to identify the most efficient allo- 
cation of funds to improve educational opportunity and success. Numerous states 
have launched endeavors to increase degree attainment (such as Tennessee’s Drive 
to 55), and college access programs are viewed as an important component of those 
efforts. 

On the whole, we know very little about the efficacy of these programs. Although 
some programs have conducted small-scale evaluations, few have done so using rig- 
orous causal methods (Maynard et al., 2014). Establishing valid counterfactuals for 
students participating in college access programs is challenging due to the selection 
bias of schools and students choosing to work with the program, a challenge we 
overcome using an experimental design. In recent years, there have been a num- 
ber of studies implementing randomized control trials to evaluate college access 
programs (Avery, 2013; Berman, Bos, & Ortiz, 2008; Carrell & Sacerdote, 2016; 
Oreopoulos, Brown, & Lavecchia, 2017; Phillips & Reber, 2018); however, they all 
examine within-school effects of programs targeting specific students within a pop- 
ulation. While these studies are greatly informative about the efficacy of student- 
targeted interventions, the diversity of college access programs necessitates further 
consideration of the efficacy of programs targeting an entire school. Our study is the 
first experimental analysis of a whole-school college access program model in the 
United States. Examining a school-wide intervention dramatically reduces concerns 
of spillover effects and other within-school confounders that can limit within-school 
research designs. Additionally, our analysis is the first to test a scaled-up model of a 
whole-school intervention, and is therefore the largest study to date, incorporating 
nearly 40,000 students. 

Our primary research question is whether providing information and support 
to high school students improves their likelihood of enrolling in postsecondary 
education. Given the ubiquity of college access programs with similar goals, the 
answer to this question sheds light on the value of the investment in college access 
programs generally. To estimate causal impacts, we exploit the random assignment 
used in the expansion of a large college access program called Advise Texas (Advise 
TX). In the 2010/2011 school year, Texas piloted the program in 15 high schools. 
In the following year, the state expanded the program to nearly 120 schools. This 
expansion offered us the opportunity to randomly assign the program to high schools 
across the state, thereby avoiding selection of schools. The program is a whole-school 
model in which one college adviser (a recent college graduate) is assigned to work 
full-time in the high school to assist with college preparation and enrollment. As 
we describe, our experiment has significant limitations, and there was a significant 
lack of compliance throughout the experiment. 
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Using administrative data from the Texas Higher Education Coordinating Board 
(THECB), our experimental analysis shows mixed results. Pooled results across 
the first three years of the program reveal no overall impact of having a college 
adviser randomly placed in a high school to assist with college enrollment. De- 
spite the lack of compliance in the experiment, we are able to estimate fairly tight 
confidence intervals, which preclude large impacts of the program. However, we 
do find that enrollment outcomes improve for several groups of disadvantaged 
students that the program prioritizes. Low-income and Hispanic students expe- 
rience increases of 2 to 3 percentage points on immediate college enrollment in 
the fall after high school graduation. These effects are concentrated among two- 
year college enrollments, but results do not suggest the program shifts students 
from four-year to two-year colleges as we observe no change in four-year enroll- 
ments. In terms of later college outcomes, we find no impact on college persistence 
with relatively tight confidence intervals. While the enrollment impacts for low- 
income students persist over two consecutive high school graduating cohorts, the 
impacts for other groups attenuate in the second and third years, eventually turn- 
ing slightly negative but insignificant. There are several potential explanations for 
the attenuation that we assess. We remain most confident in our estimates of the 
program’s impact in its first year of implementation, and these estimates suggest 
that the program has a small positive impact on two-year college enrollment for 
Hispanic and low-income students but no effect on bachelor’s degree attainment 
rates. 

We structure the paper as follows. The next section provides background informa- 
tion on college advising and specific programmatic details on Advise TX. The third 
section outlines the experimental design, data, and empirical strategies. Our fourth 
section provides the empirical results, and our fifth section discusses the results and 
provides estimates of cost effectiveness. 


BACKGROUND AND INSTITUTIONAL DETAIL 


College Advising for High School Students 


Traditionally, high school students learn about college through their guidance coun- 
selor, and a long literature documents the extent to which guidance counselors are 
overwhelmed by the large numbers of students seeking assistance. In 2013, the na- 
tionwide average ratio of students to high school guidance counselors was 470:1, 
and the ratio in Texas was 462:1 (American Counseling Association, 2014). The lack 
of support has been particularly acute among low-income and minority students 
(Avery & Kane, 2004; Lee & Ekstrom, 1987). 

There have been a number of studies focused on specific mechanisms that may 
affect students’ likelihoods of attending college. Focused interventions, implemented 
through randomized control trials, demonstrate that college enrollment increases 
when students receive help with financial aid forms (Bettinger, et al., 2012), when 
students receive encouragement over the summer between high school and college 
(Castleman, Arnold, & Wartman, 2012; Castleman, Page, & Schooley, 2014), and 
when high-achieving students receive fee waivers and informational supports in 
choosing among colleges (Hoxby & Turner, 2015). 

Our focus is on more holistic programs in which an adviser provides a host of 
services and college planning. These have proven more difficult to evaluate for 
several reasons. First, college access programs are diverse in nature and size, and 
they contain varied levels of student supports, counseling, and academic help. Few 
programs are adopted at a sufficient scale to facilitate a large-scale evaluation with 
random assignment. In their systematic review of the efficacy of college advising 
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programs, Maynard and colleagues (2014) report results for many studies with only 
a few hundred students or less. 

Another problem in the evaluation of college access programs is selection bias. 
Even when programs exist on a large enough scale to facilitate evaluation, these 
programs purposefully target schools with large proportions of disadvantaged 
students, and more motivated students likely select into receiving high doses of 
the treatment. This makes rigorous evaluation that eliminates selection critical for 
measuring program effects. Even studies employing quasi-experimental methods 
to reduce selection effects offer questionable treatment estimates. Of the 18 
evaluations of broadly defined college access programs that use experimental or 
quasi-experimental designs, 11 rely on some form of matching design to estimate 
the effects of the program (Maynard et al., 2014). In nearly all cases, the random- 
ized control trials provide smaller impact estimates than the quasi-experimental 
studies, suggesting that matching techniques do not fully account for bias and that 
randomized control trial designs are necessary to accurately measure effects. 

To date, the most rigorously evaluated programs show positive, although some- 
what mixed results. A majority of studies find that advising programs often impact 
the choice of school of attendance rather than inducing students at the margin of 
college attendance to enroll. For example, one of the largest evaluations to date 
focused on Upward Bound. Upward Bound is a cohort-based model where a small 
group of disadvantaged students are targeted early in their academic careers (typ- 
ically around eighth grade) and followed throughout high school. Upward Bound 
had no impact on college enrollment although it moved a small percent of students 
from two- to four-year schools (Myers et al., 2004). Similarly, other college pro- 
grams, such as College Possible, Bottom Line, and SOURCE, steer students toward 
four-year colleges or colleges with lower dropout rates (Avery, 2013; Berman, Bos, 
& Ortiz, 2008; Castleman & Goodman, 2018). 

In Canada, Oreopoulos, Brown, and Lavecchia (2017) and Oreopoulos and Ford 
(2016) show that mentoring programs and integrating college-going information 
into the curriculum can improve college attendance among Canadian youth. Phillips 
and Reber (2018) find that virtual advising (including text-based reminders of key 
tasks and deadlines) can improve intermediary outcomes, such as taking the SAT 
and submitting financial paperwork on time, but does not increase college enroll- 
ment (although it may improve enrollment outcomes for Hispanic students who 
speak Spanish at home). Cunha, Miller, and Weisburst (2018) find that Texas GO 
Centers, which provide information and peer guidance to high school students, in- 
crease the rate of applying to and being accepted to college, but produce no overall 
effect on college enrollment, persistence, or completion. They do find, however, a 
small positive effect on four-year college enrollment among Hispanic students. 

Perhaps the most optimistic study has been the recent evaluation of the Dartmouth 
Mentoring Program (Carrell & Sacerdote, 2016). In this program, Dartmouth under- 
graduates helped students apply for college and provided $100 in incentive money to 
students who completed college applications. The program led to improved college 
attendance for women, and importantly, they found that the effects were largest for 
students who lacked parental support. 

While many of these programs focus on providing similar services (e.g., assistance 
with completing college and financial aid applications), there are some key differ- 
ences in our setting. Programmatically, most evaluated college access programs 
are cohort-based models where students are specifically targeted because of a set of 
characteristics. Students were often targeted early in their academic careers or were 
assigned based on some criteria related to students’ backgrounds and potential to 
attend college. In our context, advisers were charged with working across the entire 
set of students at the school, typically focusing most of their efforts on seniors and 
prioritizing those students who might have the least support outside of school. 


Journal of Policy Analysis and Management DOI: 10.1002/pam 
Published on behalf of the Association for Public Policy Analysis and Management 


College Guidance for All / 5 


Perhaps the most important difference relates to scale. With the exception of 
Oreopouolos and Ford (2016), who randomly assigned schools an intervention 
integrating three college information workshops into the Canadian high school 
curriculum, all of these prior interventions were done as short-term, within-school 
evaluations. Researchers typically used randomization to choose which students 
received services. As a result, the control group students attended the same schools 
and readily interacted with treated students. The potential spillovers that may 
exist in such a setting could likely downward bias any results, which may explain 
why most programs have failed to demonstrate impacts on the extensive margin 
of college attendance. Our analysis identifies treatment effects for a school-wide 
model that incorporates any within-school spillovers as part of the overall treatment 
effect. The use of a school-wide model is also advantageous in that it allows advisers 
and teachers to provide assistance to students at the classroom level without 
having to consider students’ treatment statuses. Moreover, our program focuses 
on a scaled-up model of an intervention. Many of the other programs evaluated 
to date involve aspects of the treatment that may be implausible in a scaled-up, 
whole-school setting. For example, the cash payments to students in the Dartmouth 
program may be politically or financially implausible in most settings, or the level 
of intensity in College Possible (on average 42 adviser meetings per student) may 
be unsustainable when scaled-up over a typical school, district, or state. 

Additionally, most evaluations of college access programs follow a college ac- 
cess program over a short time horizon. One of the criticisms of randomized ex- 
periments is that the comparisons fail to account for long-run adaptations that 
might be made in the presence of a newly established intervention (Shanzenbach, 
2012). These adaptations would become more present as a program increases its 
scale and could undermine the short-term results. Our study contributes to bet- 
ter understanding longer-term program impacts as we provide some evidence that 
these adaptations may have impacted efficacy two to three years after the initial 
implementation. 

Our study complements the existing literature by providing an evaluation of a 
large-scale implementation of a college access program across 111 schools including 
over 38,000 students. The program is a whole-school model and represents a more 
scaled-up version of a college access program than provided in extant literature. 
Given the randomization of schools in Texas, our study could potentially provide 
the best evidence to date on the effectiveness of similar programs, as well as provide 
valuable insight on challenges and best practices associated with college access 
programs in other states. 


Advise TX 


The primary goal of Advise TX is to raise the rates of college enrollment and comple- 
tion among low-income, first-generation college, and underrepresented high school 
students in Texas. The program employs the model of the College Advising Corps 
(CAC) which operates in 14 other states and is headquartered in Chapel Hill, North 
Carolina. The program model integrates supplementary supports in the form of a 
college adviser assigned to a specific high school. The advisers address primarily in- 
formational barriers to college enrollment, although they also assist with academic 
and financial barriers.! 


! Nationally, CAC works with sophomores and juniors to increase their likelihood of taking AP exams and 
other college preparation courses. In Texas, however, the size of the graduating cohort was sufficiently 
large that it impeded significant interaction with underclassmen. The Texas advisers described their 
efforts as “triage” for high school seniors. 
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Advise TX partners with colleges and universities in the state to recruit and train 
recent college graduates from these partner institutions to serve as full-time col- 
lege advisers in disadvantaged high schools. Advisers participate in a six-week, 
residential summer training program prior to their placement in a high school. 
The advisers serve as near-peer mentors and often have characteristics closely 
aligned with the population of students they serve at the high schools. For ex- 
ample, most advisers are themselves first-generation college graduates. Advisers 
agree to serve for one year with the option to renew for a second year. While in 
the schools, advisers work in close collaboration with guidance counselors, teach- 
ers, and administrators within their school to foster a school-wide “college-going” 
culture. 

Although advisers serve all students at the school, their work primarily focuses 
on low-income and first-generation college students who, due to a lack of informa- 
tion and misperceptions about costs and aid, are historically underrepresented in 
postsecondary education. Advisers offer direct support to students in the form of 
individual advising sessions, group sessions with students, and group sessions with 
students and parents. 

Advisers also work most closely with seniors. Across the 2012/2013 academic 
year, the Advise TX program, including Advise TX schools that were not part of 
the experiment sample, served schools with 52,425 seniors and had direct contact 
with 84 percent of them at least once. The advisers logged over 180,000 individual 
student meetings and over 110,000 group meetings with seniors across the state. 
Conditional on having contact with the advisers, seniors received an average of 3.5 
individual meetings and 2.1 group meetings. 

Typically, advisers assist seniors with the college search process, college applica- 
tion process, and financial aid process. This work can include encouraging students 
to consider a wide range of postsecondary options accounting for the fit of student 
to college, taking students on college visits, establishing time lines, applying for fee 
waivers, interpreting communications from colleges such as offers of admission and 
financial aid, and a host of other general supports as students navigate the college 
admission and enrollment process. For example, advisers assisted 52 percent of 
seniors in Advise TX schools with completing college applications and arranged col- 
lege visits with 12 percent of the seniors. As a marker of the low-income population 
with whom they work, advisers helped 20 percent and 33 percent of seniors obtain 
fee waivers for the ACT and SAT, respectively, and they helped over 33,000 students 
receive college application fee waivers. 

Advisers also work with underclassmen to encourage students to consider and 
plan for higher education and focus on specific preparatory activities such as study- 
ing for and taking the SAT or ACT. In the 2012/2013 school year, Advise TX advisers 
met with over 25,000 juniors, 6,000 sophomores, and 4,000 freshmen, representing 
a substantial portion, but not the majority, of their efforts. Advisers typically priori- 
tize meeting with students who are underrepresented in higher education, including 
underrepresented minorities, low-income students, and first-generation students. 
In Appendix Table A1, we show the average number of interactions that students in 
these groups had with advisers in 2016/2017, the year in which we have the most 
complete data.* In each case, the prioritized group had more interactions with the 
adviser. 


2 In our adviser tracking data, we can track outcomes for all three groups. In the Texas administrative 
data, we can track outcomes for underrepresented minorities and low-income students but not first 
generation students. 

All appendices are available at the end of this article as it appears in JPAM online. Go to the publisher’s 
website and use the search engine to locate the article at http://onlinelibrary.wiley.com. 
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Table 1. Descriptive statistics and balance check of student characteristics in pretreatment 
(2009/2010). 


All T-C difference 
All TX high experimental All treatment Raw difference with lottery 
schools high schools high schools T-C controls 
Std. Std. 
Variable Mean Stdev. Mean Stdev. Mean Stdev. Difference Error Difference Error 
White 0.391 0.488 0.227 0.419 0.203 0.403 —0.038 0.037 —0.036 0.024 
Black 0.134 0.340 0.181 0.385 0.225 0.418 0.071° 0.036 0.088°" 0.022 
Hispanic 0.420 0.493 0.545 0.498 0.521 0.500  —0.040 0.057 —0.060° 0.025 
Other race 0.055 0.229 0.047 0.211 0.051 0.220 0.007 0.011 0.008 0.007 
URM 0.553 0.497 0.726 0.446 0.745 0.436 0.031 0.042 0.028 0.024 
Female 0.497 0.500 0.502 0.500 0.504 0.500 0.003 0.008 0.002 0.007 
FRL 0.363 0.480 0.474 0.499 0.478 0.500 0.007 0.042 —0.014 0.021 
Age 17.181 0.611 17.200 0.614 17.170 0.596 —0.049"" 0.017 -—0.041™ 0.014 


Fall College 0.519 0.500 0.497 0.500 0.512 0.500 0.024 0.018 0.015 0.011 
Enrollment 


N 274,623 38,201 14,270 38,201 38,201 


Notes: *p < 0.10; *p < 0.05; **p < 0.01. This table uses student level data from the 2009/2010 school 
year. The regressions estimating the difference between treatment and control use clustered standard 
errors at the school level (111 schools). Treatment assignment in the first year of the treatment is used 
as measure of treatment. 


EXPERIMENTAL DESIGN & DATA 


When the Advise TX program planned its expansion after its initial pilot year, we 
collaborated with the Texas Higher Education Coordinating Board to identify and 
randomly select high schools to receive the program. The THECB identified a sam- 
pling frame of 418 high schools in the state with at least 35 percent free/reduced price 
lunch participation, less than 70 percent of graduating students attending college 
within a year, and less than 55 percent of students experiencing a “distinguished” 
college-prep curriculum. These schools were invited to apply, and 237 did so. These 
237 schools were ranked on the above three criteria as well as percent of underrep- 
resented minority and a qualitative “fit” component that was assigned a one to four 
value by Advise TX staff based on the school’s organizational capacity. All schools 
that applied were given an aggregate score based on these criteria, and the top 84 
schools were automatically selected for the program. The bottom 42 schools were 
eliminated from consideration. The remaining 111 schools were considered eligible 
for random assignment to the program and constitute our experimental sample. 

To ensure geographic diversity, we blocked on region of the state. These 111 
schools were divided into 32 geographic regions, and a lottery was held within each 
region to select treatment schools. Thirty-six schools were randomly chosen for 
treatment assignment out of the set of 111 across the regions. 

Table 1 provides descriptive statistics at the student level for demographic vari- 
ables and for the fall college enrollment outcome measured in the 2009/2010 pre- 
treatment year, the year of data used to rank the order of schools as described above. 
The first column of numbers contains means for all Texas high schools followed by 
schools in the experimental sample and then treatment schools. Given the selection 
criteria and goals of the Advise TX program, schools in the experiment have a higher 
share of minority and low-income students than all Texas high schools, but college 
enrollment rates are quite similar. 
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Table 1 also investigates balance in pretreatment covariates and the pretreatment 
college enrollment outcome across treatment and control schools. We run block con- 
trol regressions of each variable on treatment assignment and report the treatment 
coefficient and standard error in the last two rows of Table 1. There do appear to be 
differences in the racial makeup of the schools assigned to treatment, with treatment 
schools more likely to have higher percentages of black students and lower percent- 
ages of Hispanic students than control schools.* During the randomization, Advise 
TX used only the aggregate percentage of underrepresented minorities (“URM” in 
Table 1), and the treatment and control samples are balanced on this variable. Age 
also appears slightly imbalanced within blocks, as students in treated schools are 
younger by 0.04 years, corresponding to about 15 days younger. A chi-squared test 
of the joint significance of the covariates accounting for blocking and clustering 
fails to reject the null hypothesis of balance (Hansen & Bowers, 2008). Although the 
randomization within regional blocks yielded some minor differences, we control 
for these variables in our analyses below and believe the random assignment process 
produced reasonably equivalent treatment and control groups. 

We observe student-level outcome data provided by THECB for the first three 
years of the treatment (2011/2012, 2012/2013, and 2013/2014 school years), hence, 
we estimate results at the student level despite the school-level nature of the inter- 
vention, thereby creating a clustered randomized control trial. Due to randomized 
treatment assignment, we can employ the regression model below to identify the 
causal effects of having the Advise TX program assigned to a high school on indi- 
vidual college enrollment outcomes. 


Visi = Aj + XisjB +5 Treatmentis; + &jsj (1) 


Student i at school s in region j receives a value of one for the binary treatment 
variable if the student was enrolled in a high school assigned to treatment. Because 
we blocked on region, we include region fixed effects, aj. We also include available 
student-level demographic information such as gender, race, and low-income status 
as covariates to increase precision in vector Xj.j. We estimate our binary outcomes 
using linear probability models for ease of interpretation. It is debatable whether 
clustering standard errors by school or by school-by-year level is preferable. We 
choose a more conservative approach of using standard errors which cluster at the 
school level since consecutive cohorts of graduating students may be related within 
schools. The Texas administrative data from the THECB track all students who grad- 
uate from Texas public high schools into all public institutions of higher education 
within Texas. We augment the THECB data with National Student Clearinghouse 
data enabling us to track enrollments into out-of-state and private postsecondary 
institutions. 

The above analytical approach provides intent-to-treat (ITT) estimates; however, 
compliance with treatment assignment is approximately 75 percent (Table 2). Five 
schools of the 36 assigned to treatment subsequently declined to accept an adviser. 
Advise TX requires data sharing, dedicated space, and administrative oversight. 
Many schools who initially applied were unable or unwilling to comply with these 
requirements. Additionally, nine control schools received an adviser in part to make 
up for the five treatment schools that declined to participate and in part due to 


4 We had multiple blocks where all of the schools were highly polarized. For example, Block 16 had three 
schools that were either highly Hispanic or highly African-American. Blocks were designed by geography 
and were balanced by the overall share of URM students. In Block 16, we would have had imbalance in 
the proportion of black or Hispanic students no matter which school had been selected in the lottery. If 
we exclude block 16, we reduce the imbalance. 
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Table 2. Treatment compliance in year one (2011/2012). 


Panel A: School level 


Treatment Control 
received received Total 
Treatment Assigned 31 5 36 
Control Assigned 9 66 75 
Total 40 72 111 
Lottery controlled regression of treatment 0.745 
received on treatment assignment (0.072) 


Panel B: Student level 


Treatment Control 
received received Total 
Treatment Assigned 12,529 1,324 13,853 
Control Assigned 3,267 21,004 24,271 
Total 15,796 22,328 38,124 
Lottery controlled regression of treatment 0.774 
received on treatment assignment (0.070) 


Notes: For year 2011/2012, first year of treatment. Standard error is clustered at the school level in the 
student level regression. 


philanthropic financial gifts provided by funders that conditioned their gift on par- 
ticipation of specific control schools. Although we had randomly constructed a 
waitlist with the schools assigned control status, program staff violated the waitlist 
in three instances thereby undermining the randomization of the waitlist.> We fo- 
cus on intent-to-treat estimates throughout our results tables, although simple Wald 
estimators can be used to estimate the treatment on the treated (TOT) effect inflat- 
ing the intent-to-treat effects by approximately 33 percent. In Table A2, we report 
the instrumental variable estimates that provide treatment on the treated estimates 
across subgroups and cohorts for three college enrollment outcomes.° 


RESULTS 


First Year Impacts 


We report intent-to-treat results for the first year of Advise TX on any college en- 
rollment in the fall after high school graduation in Table 3. The first two columns 
present the treatment effect on the full sample with and without covariate controls; 
given the slight imbalance in the racial composition of the schools discussed above, 
we preference the estimates with covariates. We observe a statistically insignificant 
1.1 percentage point increase in college enrollment at treatment schools. We might 
not be surprised that the point estimate is small given the intervention adds only 


5 For example, after seeing Advise TX operate in some Fort Worth schools, a local donor offered complete 
funding for the program as long as Advise TX would expand into all schools, including control schools, 
in the local area. 

6 All appendices are available at the end of this article as it appears in JPAM online. Go to the publisher's 
website and use the search engine to locate the article at http://onlinelibrary.wiley.com. 
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Table 3. Intent-to-treat first-year college fall enrollment results. 


Full sample Black Hispanic Low-Income 

Treatment 0.022" 0.011 0.019 0.012 0.022+ 0.020+ 0.038°" 0.019 

(0.011) (0.010) (0.017) (0.016) (0.012) (0.011) (0.014) (0.012) 
Covariates Yes Yes Yes Yes 
Block Fixed Yes Yes Yes Yes Yes Yes Yes Yes 

Effects 

Control Mean 0.558 0.599 0.515 0.498 
R? 0.022 0.090 0.009 0.064 0.038 0.085 0.025 0.083 
N 38,124 38,124 6,659 6,659 21,852 21,852 19,677 19,677 


Notes: *p < 0.10; *p < 0.05; **p < 0.01. Each cell reports the coefficient on treatment assignment in 
2011/2012 for each sample using a linear probability model. Standard errors are reported in parentheses 
and are clustered at the school level. Covariates include gender, race, age, whether the student was on 
free/reduced price lunch, whether free/reduced price lunch was missing, and whether the entire school 
was on free/reduced price lunch. 


one college adviser in an entire high school. One noisy estimate of the effect of 
an additional high school counselor on college enrollment suggests an additional 
counselor might increase college going by 10 percentage points (Hurwitz & Howell, 
2014), but those results appear to be driven by increasing from one to two coun- 
selors at very small schools. Schools in our sample are, on average, over four times 
as large likely resulting in a diluted effect. We further consider effects by school size 
below. 

Given the program’s goals and the prioritized populations, advisers may have 
a larger effect on specific subgroups. The subsequent columns of Table 3 report 
treatment effect estimates on minority and low-income subsamples. We observe 
positive but insignificant point estimates for black students and a 2 percentage point 
effect on Hispanic students that is significant at the 10 percent level in the model 
with covariates. We observe a similar effect for low-income students of 2 percentage 
points, but with a p-value of 0.103. Overall, we conclude that the program likely had 
a | to 2 percentage point effect on college enrollment in its first year, concentrated 
among Hispanic and low-income populations, although we do not have enough 
power to precisely estimate effects smaller than 2 percentage points. 

This overall enrollment effect masks important differences in enrollment patterns 
across institutions. Table 4 displays the intent-to-treat estimates for fall college en- 
rollment outcomes separated by two-year and four-year college enrollment. In the 
full sample and across all three subgroups, we observe larger treatment effects on 
enrollment at two-year institutions than at four-year institutions. Overall, the pro- 
gram increased two-year college enrollment by 2.4 percentage points in its first year 
with larger effects for Hispanic students of 3.4 percentage points and marginally 
significant effects of 2 percentage points for low-income students. Given the two- 
year enrollment rates of control students, these estimated effects correspond to a 6.3 
percent increase for the full sample, a 9.1 percent increase for Hispanic students, 
and a 5.5 percent increase for low-income students. In contrast, we see no evidence 
of effects for black students and no movement in four-year college enrollment rates 
with point estimates close to zero in each sample. The program’s overall college 
enrollment effects are driven by increases in two-year college enrollment, and, im- 
portantly, these effects do not appear to be at the cost of four-year enrollments. The 
program improves college enrollment rates for students at the margin of two-year 
college attendance without shifting students away from four-year colleges. We note 
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Table 4. Intent-to-treat first-year college fall enrollment results for two-year versus four-year 
enrollment. 


Two-Year enrollment Four-Year enrollment 


Full Low- Full Low- 
sample Black Hispanic Income sample Black Hispanic Income 


Treatment 0.024" 0.009 0.034" 0.020* —0.007 0.006 —0.006 0.006 
(0.012) (0.019) (0.013) (0.012) (0.010) (0.016) (0.009) (0.010) 
Covariates Yes Yes Yes Yes Yes Yes Yes Yes 
Block Fixed Effects Yes Yes Yes Yes Yes Yes Yes Yes 
Control Mean 0.380 0.383 0.374 0.365 0.237 0.277 0.184 0.175 
R2 0.034 0.016 0.041 0.038 0.077 0.057 0.060 0.055 
N 38,124 6,659 21,852 19,677 38,124 6,659 21,852 19,677 


Notes: tp < 0.10; *p < 0.05; **p < 0.01. Each cell reports the coefficient on treatment assignment in 
2011/2012 for each sample using a linear probability model. Standard errors are reported in parentheses 
and are clustered at the school level. Covariates include gender, race, age, whether the student was on 
free/reduced price lunch, whether free/reduced price lunch was missing, and whether the entire school 
was on free/reduced price lunch. 


that this may imply impacts on students not at the top of the academic prepara- 
tion distribution, although we do not directly observe any academic performance 
measures in our data. 

We note that although we did not see a statistically significant difference in pre- 
treatment college enrollment effects, the pretreatment point estimate for treatment 
schools is approximately 1.5 percentage points higher than for control schools. If 
this point estimate is indicative of a real pretreatment difference that remains stable 
in the subsequent treatment years, it could account for much of the positive effect 
we observe. It would suggest the treatment effects we observe may be overestimates, 
and that the true enrollment effect is closer to 1 percentage point for overall two-year 
enrollments with higher rates among Hispanic students. 

Given the lack of compliance to treatment assignment noted in Table 2, the treat- 
ment effects reported above are larger for schools that actually had an adviser 
working in the school. As observed in Table A2, the two-year college treatment on 
the treated effects for Hispanic students and low-income students in the first cohort 
is 4.5 and 2.4 percentage points, respectively.’ 


College Application and Persistence Outcomes 


We now consider two other observable and pertinent outcomes. Advise TX uses the 
number of college applications submitted by each student as a performance measure 
under the assumption that the college adviser will improve the college-going culture 
of the school and directly assist students with completing college applications. We 
observe the number of college applications submitted by each student to any public 
institution of higher education in Texas and assess whether the advisers increase 
the number of college applications to these institutions in panel A of Table 5. 

We do not observe a large effect on the number of applications. The only 
marginally significant result exists for low-income students, and the effect is small 


7 All appendices are available at the end of this article as it appears in JPAM online. Go to the publisher's 
website and use the search engine to locate the article at http://onlinelibrary.wiley.com. 
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Table 5. First-year intent-to-treat estimates on college application and persistence outcomes. 


Panel A: College application outcome 


Full sample Black Hispanic Low-Income 
Number Binary Number Binary Number’ Binary Number Binary 
ofapps applied ofapps applied ofapps applied ofapps applied 
Treatment 0.031 0.013 0.124 0.024 0.032 0.021+ 0.099+ 0.031" 
(0.050) (0.010) (0.088) (0.020) (0.054) (0.011) (0.053) (0.011) 
Control Mean 0.646 0.601 0.813 0.654 0.589 0.559 0.586 0.537 
Covariates Yes Yes Yes Yes Yes Yes Yes Yes 
Block Fixed Effects Yes Yes Yes Yes Yes Yes Yes Yes 
R? 0.061 0.090 0.074 0.079 0.050 0.094 0.067 0.080 
N 38,123 38,124 6,659 6,659 21,851 21,852 19,676 19,677 
Panel B: College persistence outcome 
Full Low- 
sample Black Hispanic Income 
Treatment 0.006 0.010 0.009 0.017 
(0.010) (0.017) (0.011) (0.012) 
Control Mean 0.398 0.380 0.364 0.334 
Covariates Yes Yes Yes Yes 
Block Fixed Effects Yes Yes Yes Yes 
R? 0.081 0.064 0.067 0.071 
N 38,124 6,659 21,852 19,677 


Notes: *p < 0.10; *p < 0.05; **p < 0.01. Each cell reports the coefficient on treatment assignment for 
each outcome and for each sample. The number of applications is measured continuously. The binary 
applied outcome is an indicator for applying to at least one institution. College persistence is a binary 
measure of whether students were enrolled in a second year of higher education. Binary outcomes are 
measured using linear probability models. Standard errors are reported in parentheses and are clustered 
at the school level. Covariates include gender, race, age, whether the student was on free/reduced price 
lunch, whether free/reduced price lunch was missing, and whether the entire school was on free/reduced 
price lunch. 


at a tenth of an application. This implies the program induced one out of every 10 
low-income students to apply to an additional college. This is not overly surprising 
given that Texas has a common application, and, conditional on having applied 
to one college, applying to an additional college literally involves only a click of a 
button. 

We also examine whether the program affected students at the margin of applying 
to any college using a binary measure of applying to higher education. Here, we 
observe stronger results with a 2 percentage point effect among Hispanic students 
and a 3 percentage point effect among low-income students. Advisers are motivating 
some students who would not otherwise apply to college to take a major step toward 
enrolling. These findings comport with the small increases we observe in two-year 
college enrollment among Hispanic and low-income students. 

Although advisers focus on college enrollment, they may improve the fit or 
“match” between students and institutions through the advising process. This im- 
proved match may result in increased persistence (Bowen, Chingos, & McPherson, 
2009); therefore, we also examine college persistence outcomes as a test for this im- 
proved match hypothesis and report results in panel B of Table 5. Across all of the 
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Table 6. Intent-to-treat college fall enrollment estimates pooled and in program years one, 


two, and three. 


Full sample Black Hispanic Low-Income 
Pooled Years 0.006 0.000 0.009 0.005 0.004 0.004 0.029"** 0.015 
(0.009) (0.008) (0.014) (0.012) (0.011) (0.010) (0.011) (0.010) 
0.542 0.568 0.506 0.487 
2011/2012 (Year 1) 0.022* 0.011 0.019 0.012 0.022+ 0.020+ 0.038" 0.019 
(0.011) (0.010) (0.017) (0.016) (0.012) (0.011) (0.014) (0.012) 
Control Mean 0.558 0.599 0.515 0.498 
2012/2013 (Year2) 0.013 0.006 0.025 0.018 0.009 0.009 0.046" 0.030" 
(0.010) (0.009) (0.017) (0.016) (0.013) (0.012) (0.013) (0.011) 
Control Mean 0.561 0.588 0.526 0.502 
2013/2014 (Year 3) 0.014 0.016 0.015 0.015 0.014 0.012 0.005 —0.004 
(0.011) (0.010) (0.017) (0.016) (0.013) (0.011) (0.012) (0.011) 
Control Mean 0.510 0.520 0.480 0.462 
Covariates Yes Yes Yes Yes 
Block Fixed Effects Yes Yes Yes Yes Yes Yes Yes Yes 


Notes: tp < 0.10; *p < 0.05; **p < 0.01. Each cell reports the coefficient on treatment assignment for each 
year and for each sample using a linear probability model. Standard errors are reported in parentheses 
and are clustered at the school level. Covariates include gender, race, age, whether the student was on 
free/reduced price lunch, whether free/reduced price lunch was missing, and whether the entire school 
was on free/reduced price lunch. R? and sample size vary by year and sample. 


samples, we observe small, positive, and statistically insignificant results. Although 
the results reported in Table 5 are unconditional on college enrollment, condition- 
ing on college enrollment does not change the conclusion that there is no evidence 
that having a college adviser in your high school improves your institutional match, 
as far as that match results in increased persistence.® 

Perhaps we should not be surprised by the null findings on number of applications 
and college persistence as all of the observed enrollment effect applies to two- 
year colleges. Students typically do not apply to more than one two-year college. 
Furthermore, the average persistence rates at two-year colleges are generally lower 
than at four-year colleges, so any matching benefit the advisers may be achieving 
might be countered by lower persistence among students induced to attend two- 
year colleges. Regardless of the reason, we do not observe any significant impacts 
on persistence in Table 5. 


Treatment Effects over Time 


Thus far, we have reported results for the initial year of the program, the 2011/2012 
academic year. We have two subsequent years of data for the 2012/2013 and 
2013/2014 academic years. We report intent-to-treat effects of the three pooled years 
and the separate years of program implementation in Table 6 (we replicate first year 
treatment effects from Table 3 for comparison). We note that the pooled results re- 


8 Our data do not allow us to track students beyond the second year of college enrollment. Using the 
NSC data (which tends to understate enrollments relative to the state data), we also find no significant 
impacts on persistence after the first year. As in the impacts in Table 5, we lack power to detect extremely 
small impacts. 
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veal no statistically-significant program impacts when controlling for our preferred 
model including covariates in either the full sample or for any subgroup. However, 
these pooled results mask important differences in treatment effect estimates over 
time. Focusing on the full sample results, we observe the positive 2 percentage point 
treatment effects observed in year one falling to an insignificant 1 percentage point 
effect in year two and declining to a negative point estimate in year three of the 
program. This pattern generally holds for each subgroup, with the treatment effect 
declining over time such that there are only null or negative treatment effects by 
the third year of the program. We consider several possible explanations for this 
reduction in treatment effect over time below. 

The declining treatment effects over time present a puzzle, and we consider four 
separate hypotheses that may explain the pattern of results. The first potential ex- 
planation focuses on treatment compliance. Even in the first year, compliance was 
only 75 percent (Table 2). This was due to five initially assigned treatment schools 
backing out of their commitment and not accepting an adviser, combined with nine 
control schools receiving an adviser. Compliance continued to deteriorate over the 
subsequent years, with some schools leaving the treatment and some control schools 
receiving an adviser as the program expanded.’ By the 2015/2016 school year, 20 of 
the 36 schools initially assigned treatment had left the treatment and did not have 
an adviser, and 17 of the initial 75 control schools had an adviser. This eroding com- 
pliance over time dilutes the treatment contrast, possibly leading to the attenuation 
of the effects we observe. 

However, when we examine treatment on the treated effects by instrumenting for 
treatment receipt with treatment assignment (Table A2), we observe similar patterns 
of positive effects for Hispanic and low-income students in the first year declining 
to no effects in year three.!° The decline in the TOT results are roughly aligned with 
the decline observed in the ITT results suggesting that adjusting for compliance does 
not substantially explain the falling estimates. 

A second hypothesis as to why the treatment effect fade-out occurred involves the 
size of schools in compliance with treatment assignment. A reasonable hypothesis 
is that size is a mediating factor in the treatment. One adviser in a school of 50 
graduating seniors can work with a higher fraction of the student body, and at a 
higher level of dosage, than a school with a graduating class of 700. If smaller schools 
experience a larger treatment effect and if smaller treatment schools are less likely 
to maintain an adviser over the treatment period, we might expect the treatment 
effect to decline over time. In the distribution of treatment school size, the median, 
75th percentile, and 90th percentiles of schools who had an adviser did not change 
over time. By contrast, the 10th and 25th percentiles changed as the program moved 
towards large schools, so compliance is related to school size. The treatment effect 
might be greater at smaller schools because our dependent variable is measured in 
enrollment rates. If any individual counselor can affect a fixed number of students, 
then the impact on rates will be smaller in schools with larger student bodies. We 
also have descriptive evidence that advisers in smaller schools can devote more time 
to each student. In one small treatment school, 90 percent of students met with an 


° Evidence from supplementary qualitative work documented that schools left the treatment conditions 
in subsequent years for different reasons (Bettinger et al., 2014). In several cases, the university partner 
cancelled the program because the schools were too distant from the institution. In one case, the lead 
guidance counselor did not want to renew the adviser because of poor performance. In another case, 
there was a competing college access program in the school, and the adviser left to work in that competing 
program. 

° All appendices are available at the end of this article as it appears in JPAM online. Go to the publisher’s 
website and use the search engine to locate the article at http://onlinelibrary.wiley.com. 
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adviser over 10 times, while the average number of meetings is 3.5 meetings per 
student across all treated schools. 

However, when we estimate the treatment impact interacted with size, we find 
few significant results. We estimate differential size effects using several alternative 
specifications (linear size, log size, and binary indicators for small schools), but we 
find little evidence of differential impact (Table A3).'' Although we lack statistical 
power to fully identify how size interacts with treatment, the lack of significance 
does not support the hypothesis that size is the primary mechanism of explaining 
the fade-out of treatment effects over time. 

A third possible reason for the decline of treatment effects is that control schools 
adopted substitutes for the Advise TX program. While we have documented the com- 
pliance with respect to Advise TX, we have not documented the existence of other 
college access programs. Control schools not assigned the program may adopt an ef- 
fective alternate program thereby reducing the experimental treatment effect. While 
potentially serving as a good test of general equilibrium for widespread program 
adoption, this possible substitution among control schools potentially undermines 
identifying the program-specific effects in our experiment. Unfortunately, we do 
not have a comprehensive list pre- and post-treatment in the number and types of 
college advising programs available in control and treatment schools. However, in 
2015/2016, we conducted a survey inventory of programs at treatment and control 
schools. We asked schools to provide a year-by-year accounting of the availability 
of a wide array of common college access programs (e.g., AVID, TRIO, GEAR UP) 
hosted at the school between 2010/2011 and 2015/2016.'* Although 35 of the 36 
treatment schools provided data, only 69 percent (52 of 75) of the control schools 
replied. Our data are not completely representative of the control sample; yet, exam- 
ining the number of programs available each year provides a sense of how control 
schools responded to not being assigned the Advise TX program. 

Examining the number of programs active over time reveals suggestive evidence 
of the substitution pattern. Figure 1 provides the number of programs (including 
Advise TX) in each academic year active at treatment and control schools. In the last 
pretreatment year in 2010/2011, the number of programs is very similar between as- 
signed treatment and control schools with a statistically insignificant 0.35 program 
difference. In the first two years of treatment, we see the treatment schools growing 
in the number of programs relative to control schools due to the adoption of Advise 
TX. However, after the 2012/2013 academic year, we observe the number of pro- 
grams in control schools increasing while the number in treatment schools remains 
constant. By the 2014/2015 academic year, the difference in number of programs 
between treatment and control schools is less than in the first year of treatment. 
Hence, there appears to be some empirical support for the substitution hypothesis. 

A fourth hypothesis is that schools made within-school staff adjustments in re- 
sponse to program adoption. Treatment schools may divert guidance counselor 
resources away from college preparation and information activities once an Advise 
TX adviser arrives in the school. We again rely on the treatment and control school 
surveys to evaluate this hypothesis by examining schools’ responses to the question: 
“When a student comes to a guidance counselor with a college related question, 
what is the typical process for handling that question at your school?” Forty percent 
of schools that were assigned treatment answer that the student is referred to a spe- 
cific guidance counselor who oversees college-going activities, but only 25 percent 


'l All appendices are available at the end of this article as it appears in JPAM online. Go to the publisher's 
website and use the search engine to locate the article at http://onlinelibrary.wiley.com. 
!2 Schools could also report additional college advising programs not explicitly named on the survey. 
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Figure 1. Number of College Advising Programs Available in Treatment and Control 
Schools over Time. [Color figure can be viewed at wileyonlinelibrary.com] 


of control schools respond in this way. This 15 percentage point difference may 
be a result of guidance counselors at treatment schools reducing their workload 
on college-related activities by sending them to the Advise TX adviser. Observing a 
potential change in this measure over time would provide stronger support for this 
hypothesis, but in the absence of such longitudinal data, the cross-sectional treat- 
ment and control contrast provides suggestive support for the staffing adjustment 
hypothesis. 

The potential for staff adjustments has important implications. One criticism of 
randomized experiments is that the underlying treatment changes as individuals 
and organizations adjust to the treatment. These general equilibrium-type effects 
could cause the short-run impacts to overstate the impact of a program in the short- 
run compared to its impact when conducted at scale. In particular, Advise TX was 
intended to be a complement to existing services, but we have some suggestive 
evidence that it might have become a substitute over time. Such a shift in the 
program’s intent and interaction with existing services also has broad implications 
for other randomized experiments where the treatment may vary once scaled.!* 

In summary, it appears likely that the observed drop in positive treatment effects 
over time is due to a combination of annually declining compliance to treatment as- 
signment, additional college advising program adoption in control schools, and sub- 
stitution of resources away from college advising in treatment schools. These issues 
speak to the importance of monitoring compliance throughout the course of longi- 
tudinal experimental analyses. They also suggest general limitations to the clustered 
design of the experiment: It is difficult to observe compliance within schools and 
difficult to identify heterogeneous program effects by school characteristics given 
the relatively small number of randomized units. In contrast, student-level random- 
ization would have provided substantially more power and would have enabled a 
school-to-school comparison of effects. However, such a research design is impos- 


'3 Throughout the research, we regularly informed Advise TX on the nature of the results. Upon becoming 
aware of the possible decrease in school services, Advise TX modified their school contracts to include a 
clause about continued effort by school guidance counselors. The responsiveness of Advise TX to these 
findings provides some evidence on the potential of researcher-practitioner partnerships. 
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sible for a program designed to function at the school level such as Advise TX and 
would have introduced substantial risk of within-school spillover effects, a concern 
we have with prior within-school randomization evaluations of college advising. 


CONCLUSION 


Our results lead us to conclude that the program improved two-year college en- 
rollment rates, mostly for Hispanic and low-income students, which supports the 
general findings in the literature that providing information and college assistance 
improves college enrollment. Our results also extend the current literature by ex- 
perimentally establishing that a holistic, school-wide college access program im- 
proves postsecondary enrollment outcomes for underserved high school students. 
In contrast to several prior experimental studies that have targeted interventions 
on selected students, we demonstrate that a school-wide program can have positive 
enrollment effects at the margin of attendance. The effects we observe are substan- 
tially smaller than those reported by Carrell and Sacerdote (2016), but that is likely 
explained by the difference in targeting and dosage in their intervention. At their 
largest high schools, roughly 30 students were treated by a team of advisers for mul- 
tiple hours each week for several weeks. Combined with financial incentives, this 
intervention increased college enrollment rates for women by 14 to 15 percentage 
points. This effect is substantially larger than our observed effect of a few percent- 
age points on college attendance, but the Advise TX intervention targeted the entire 
school with one adviser working with many times the number of graduating high 
school students and investing in the preparation activities of underclassmen. The 
targeted intervention studied by Carrell and Sacerdote also relies on pre-identifying 
students at the margin of attending college. If that pre-identification is correct, their 
program may be more efficient. If, however, there is error in that pre-identification, 
students who need the intervention may be overlooked. Although the Advise TX 
program may waste resources on students who would have otherwise attended col- 
lege or who have a very low probability of attendance even with assistance, it likely 
reaches a higher percentage of students who can be switched to a postsecondary 
enrollee. 

Our results are more aligned with those of Oreopoulos and Ford (2016) who exam- 
ine a school-wide intervention focused on incorporating assistance with completing 
college applications into the senior high school curriculum in Canada. They find the 
intervention increased two-year college enrollments by 5 percentage points, only 
slightly larger than our treatment on the treated effects, albeit in a very different 
international context. 

Results from our large-scale, state-wide analysis of the Advise TX college access 
program provide important considerations for policymakers. Our results provide 
evidence in cautious support of continued funding at the federal, state, and insti- 
tution levels towards college access programs targeting low-income and minority 
populations, where we observe the largest two-year college enrollment results. The 
lack of observed positive effects on four-year enrollments and persistence, however, 
suggests programs may need to focus renewed efforts on promoting baccalaureate 
degree attainment, financial aid to enable students to afford four-year colleges, and 
better match to improve persistence. 

The results also suggest current college access programs should reflect on their 
program design. While we cannot definitely attribute effects of Advise TX to the 
near-peer design of the intervention, other qualitative work on the impact of Advise 
TX suggests that this plays an important role (Bettinger et al., 2014). They find that 
students seemed “connected” with advisers because of the fact that the advisers had 
recently been in similar high schools and were closer to their age. 
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Based on our analysis of the decline of effects over time, we also encourage 
programs to consider how their intervention can supplement rather than supplant 
existing college advising services. Funders and programs can solicit guarantees from 
schools that program adoption will not lead to a replacement of established college 
advising activities, and schools can monitor how their established college advising 
services may change after program adoption. 

Of special interest is the extent to which the program covers its costs. If we accept 
the pooled three-year treatment effect estimates of no impact, then the program 
would not be cost-effective. However, given our potential explanations of the at- 
tenuating effects over time, the cost-benefit analysis is more nuanced. If schools 
are substituting guidance counselor efforts with the Advise TX adviser, then any 
benefit of the program observed in its first year of implementation falls to zero. If, 
however, the treatment effect diminishes due to control schools adopting additional 
programs, then the program still provides a benefit. Under this scenario, it is valu- 
able to consider the cost-benefit analysis given the first-year treatment effects. We 
provide such a detailed cost-benefit analysis in Table A4 and summarize it here.'* 
Our point estimate on the effect of the treatment on two-year college enrollment 
(Table 4, column 1) suggests that 11.1 additional students per high school attended 
community college as a result of Advise TX. Given our lack of finding of any sig- 
nificant impact on persistence, we make the assumption that all of these students 
acquired just one year of college. 

If indeed the program did not lead to differential attendance except in the first 
year, then 11.1 students acquired “some college” and nothing more. If we use the 
College Board (2016) earnings for some college, it suggests that earnings are $4,900 
higher per year than they would have been with only a high school diploma. Because 
that estimate likely incorporates some students at four-year colleges who dropout 
after two or three years, we take a more conservative approach, by cutting this esti- 
mate of the benefits of some college by half.'> This suggests that earnings increased 
by $27,165 in each school after the first year that students attended just one year of 
college. 

The biggest cost to Advise TX is not the adviser costs. The average cost for the 
advisers is roughly $59,000 per school which includes salary and overhead. The 
largest cost is the foregone income from attending college. The 11.1 students who 
now attend college forego some earnings. Using College Board data (on returns to 
high school) and the National Center for Education Statistics (wages of currently 
enrolled college students), we estimate that each student foregoes almost $21,000 per 
year. This is likely an overestimate given the high unemployment rates of high school 
graduates who do not attend college in the years just after graduation, but it serves as 
a conservative estimate. As students drop out of college and join the workforce, these 
foregone wages decline and students now start experiencing some of the returns to 
college. We estimate that community college tuition and fees balance out with state 
and federal need-based financial aid programs given that most of the impact occurs 
with free/reduced lunch students. 

We can then compute the lifetime increase in earnings by combining the costs 
and benefits. If we assume some college completion, with a conservative 5 percent 
discount rate, we compute that average lifetime gains per school per year is nearly 
$156,000. As long as the returns to one year of community college exceed $1,570 


14 All appendices are available at the end of this article as it appears in JPAM online. Go to the publisher’s 
website and use the search engine to locate the article at http://onlinelibrary.wiley.com. 

'5 This estimate is even more conservative than the data in Texas would suggest. Using Texas data 
from the American Community Survey, the annual earnings differential for less than one year of college 
relative to a high school diploma is $5,407. 
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per year, then the program generates positive returns given the number of students 
who are impacted and the low cost per school. 

We view these as conservative estimates for a variety of reasons. First, we have 
been liberal in estimating the foregone wages. High young adult unemployment and 
low entry wages should lower the foregone wages. Second, we have assumed that 
CAC does not affect time to degree, subsequent return to college, or other long-run 
outcomes. Finally, we also ignore any non-pecuniary benefits of college that would 
likely improve the returns for college (Oreopoulos & Petronijevic, 2013). If we lift 
any of these assumptions, the estimated return swells. 

Even with our most conservative estimates, we find an internal rate of return of 
just over 4 percent. We know of no other estimates of the return to college access 
programs. Our result is similar or better than the return to financial aid models. For 
example, Dynarski (2008) finds a 9 percent rate of return for Georgia Hope. Other 
financial aid programs such as the Ohio College Opportunity Grant (Bettinger, 2015) 
suggest returns that are closer to 1 percent. 

In conclusion, results from our cluster randomized control trial provide the first 
causal evidence of an at-scale school-wide college access program. The evidence is 
mixed. While the program does not generate significant overall impacts on enroll- 
ment, the program is effective at increasing first-year college enrollment for low- 
income and Hispanic students, inducing them to enroll in two-year colleges when 
they otherwise would not enroll in college immediately after high school graduation. 
However, the impacts do not persist beyond the first year of implementation, and 
we do not find impacts on persistence after initial enrollment for the first cohort. 
Despite issues of compliance in the experiment, our estimates are relatively precise, 
allowing us to preclude large impacts even when the estimated impacts are statis- 
tically insignificant. Even so, given the low cost of the program per school, even a 
modest improvement in wages results in a rate of return conservatively estimated 
at 4 percentage points. These findings provide empirical support for policymakers 
choosing to continue investment in college access programs generally, although 
further analysis of different program structures and comparisons between whole- 
school interventions versus targeting individual students is a promising area for 
further research. 
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APPENDIX 


Table A1. Average number of adviser interactions by student characteristics. 


# of # of # SAT 
Student # of 1:1 group/class parent # assist on registrations 
Characteristic meetings meetings meetings applications assisted 
Free/Reduced 5.1 3.9 0.3 2.3 1.0 
Lunch 
Not FRL 3 2.2 0.1 0.8 0.2 
First Generation 6.5 4.4 0.5 3.3 1.1 
Not First 5.1 2.6 0.1 1.8 0.5 
Generation 
Hispanic 4.7 3.9 0.3 2.2 0.9 
Black 4.8 3.2 0.2 1.9 0.6 
White 2.7 2.0 0.1 0.8 0.3 


Notes: Data focus on all interactions in the 2016/2017 school year in Advise TX schools. This is the first 
year where complete student-level reporting was possible on race, income, and first generation status. 
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Table A2. Treatment on the treated instrumental variable estimates. 


Panel A: Fall college enrollment 


Full sample Black Hispanic Low-Income 
2012 Treatment 0.014 0.015 0.026+ 0.023* 
(0.012) (0.0207) (0.015) (0.014) 
2013 Treatment 0.008 0.024 0.012 0.038" 
(0.012) (0.020) (0.016) (0.013) 
2014 Treatment —0.025* —0.027 —0.017 —0.005 
(0.015) (0.030) (0.016) (0.015) 


Panel B: Fall two-year college enrollment 


Full sample Black Hispanic Low-Income 
2012 Treatment 0.031* 0.012 0.045" 0.024+ 
(0.016) (0.025) (0.019) (0.014) 
2013 Treatment 0.013 0.000 0.023 0.026+ 
(0.015) (0.021) (0.019) (0.014) 
2014 Treatment 0.008 —0.035 —0.002 —0.007 
(0.014) (0.023) (0.016) (0.013) 


Panel C: Fall four-year college enrollment 


Full sample Black Hispanic Low-Income 

2012 Treatment —0.009 0.008 —0.007 0.008 

(0.012) (0.021) (0.013) (0.012) 
2013 Treatment —0.003 0.030 —0.008 0.012 

(0.011) (0.019) (0.010) (0.010) 
2014 Treatment —0.016* 0.010 —0.015 0.002 

(0.009) (0.018) (0.011) (0.009) 
Covariates Yes Yes Yes Yes 
Block Fixed Effects Yes Yes Yes Yes 
N 38,124 6,659 21,852 19,677 


Notes: tp < 0.10; *p < 0.05; **p < 0.01. Each cell reports the coefficient on treatment received as 
instrumented using treatment assignment in 2011/2012 for each sample using a linear probability model 
to estimate the binary outcome. Standard errors are reported in parentheses and are clustered at the 
school level. Covariates include gender, race, age, whether the student was on free/reduced price lunch, 
whether free/reduced price lunch was missing, and whether the entire school was on free/reduced price 


lunch. 
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Table A3. Treatment effect heterogeneity by school size. 


2012 Fall college 2012 Fall two-year 2012 Fall four-year 
enrollment enrollment enrollment 


Size = Number of students/1,000 


Treatment 0.010 0.036 —0.034 
(0.027) (0.033) (0.027) 
Size 0.037" 0.040™* —0.003 
(0.010) (0.012) (0.009) 
Treatment* Size —0.002 —0.008 0.013 
(0.012) (0.014) (0.013) 


Size = Ln(number of students) 


Treatment —0.026 —0.026 —0.071 
(0.168) (0.202) (0.157) 

Size 0.054" 0.054" 0.003 
(0.019) (0.021) (0.015) 

Treatment* Size 0.005 0.006 0.009 
(0.022) (0.027) (0.021) 


Size = Small school indicator for below median 


Treatment 0.009 0.025 —0.011 
(0.014) (0.016) (0.015) 

Size —0.031" —0.031* 0.000 
(0.016) (0.017) (0.013) 

Treatment* Size 0.002 —0.006 0.010 
(0.022) (0.026) (0.021) 


Size = Small school indicator for below 25th percentile 


Treatment 0.017 0.038" —0.014 
(0.011) (0.013) (0.012) 
Size —0.036° —0.022 —0.022 
(0.017) (0.019) (0.013) 
Treatment*Size —0.035 —0.072° 0.025 
(0.024) (0.032) (0.021) 
N 38,124 38,124 38,124 


Notes: *p < 0.10; *p < 0.05; **p < 0.01. Each model includes covariates and blocking fixed effects. 
Covariates include gender, race, age, whether the student was on free/reduced price lunch, whether 
free/reduced price lunch was missing, and whether the entire school was on free/reduced price lunch. 
Standard errors are reported in parentheses and are clustered at the school level. 
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Variable Values Source 
Estimated impact on college 0.024 Table 4, column 1 
attendance 
Average class size (Senior class) 462 Table 1, panel A, “Total Students” in 
Treatment HS 
Total students impact per school 11.1 Impact* Average Class Size 
Increase in earnings for BA degree $24,600 College Board, Education Pays, 2016, 
Figure 2.1 
Increase in earnings for Associate’s $9,200 College Board, Education Pays, 2016, 
degree Figure 2.1 
Increase in earnings for any college $2,450 College Board, Education Pays, 2016, 
Figure 2.1. Assumes 1/2 of the 
“some college” return for more 
conservative estimate 
Proportion of completing an AA/AS 0.30 Shapiro et al., 2016 (Figure 12) 
conditional on 2-yr start 
Proportion of completing a BA/BS 0.09 Shapiro et al., 2016 
conditional on 2-yr start 
Proportion still enrolled 0.16 Shapiro et al., 2016 
Total students who achieved AA/AS 0.0 Impact*Total Students 
Total students who received a BA/BS 0.0 Impact*Total Students 
Total students still enrolled 0.0 Impact*Total Students 
Still enrolled at 2-year 0.0 Breaking up still enrolled using 
Degree ratio from Shapiro et al., 
2016 
Still enrolled at 4-year 0.0 Breaking up still enrolled using 
Degree ratio from Shapiro et al., 
2016 
Students who received any college 11.1 Remaining students 
Average gain in earnings for students $27,165 Total students in each category 
at school multiplied by increase in earnings 
Years of college 1 
Foregone earnings in any year $20,948 Difference between high school 
earnings and NCES estimate of 
earnings while enrolled 
Total gain in wages (5 percent $156,195 PDV of earnings (starting in year 7) 


discount rate) 


less foregone wages during college 
and cost of the program 
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